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ABSTRACT 

This presentation focuses on an ongoing instructional innovation research and development 
project centered around the development of a blended, online and face-to-face introductory 
engineering graphics course. The work presented here is an in-depth analysis of how students 
make use of the online resources to supplement the instructional support they receive in class. 
The researchers were particularly interested in answering questions concerning not only what 
resources were accessed, but in what order, and whether there is any statistical correlation to 
learning outcomes. In this study, the focus was on resources related to the textbook materials 
and quizzes and tests associated with this material. The data being analyzed was collected 
from 180 students taught by two different instructors over one semester. Background on the 
project, analysis of Moodle log file data, along with recommendations for further refinement of 
instructional strategies will be presented. 

Keywords: blended instruction, online assessments, log file analysis. 


INTRODUCTION 


The movement to online education using computer-based systems has been well underway for 
over 10 years [1]. Among other reasons, higher education administration has sensed the potential of 
budgetary savings in the delivery of material online that has historically been delivered in lectures 
[2]. More importantly, the use of information-computer technologies (ICT) as a means to broaden 
participation on higher education and support 21st century skills [3] has only accelerated the pace 
of development of online learning opportunities in post-secondary education. Not surprisingly, 
engineering education has been among the areas pursuing online course offerings even though 
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practitioners and researchers have also noted the particular challenges of providing online instruc¬ 
tion for curriculum that has a large laboratory component [4]. 

To address some of the difficulties of repurposing all of a course’s content online, a method chosen 
by many engineering instructors is a blended, or hybrid, style of course offering where some of the 
course material that was originally delivered in a traditional face-to-face setting is instead delivered 
online [5]. Some of the course content, however, continues to be provided in a face-to-face model 
of instruction. In such an approach, lab-based material that is hard to transform into online content 
can continue to be taught in a face-to-face, synchronous manner. A hybrid format provides flexibility 
in delivery and format that at times can be hard to match with comparable traditional methods [6]. 
The asynchronous nature of web-based materials means that much of it can be viewed anytime, 
anyplace, and as many times as desired [7]. It follows that students often express a preference 
for this style of delivery [8]. In addition to distributing instructional content between online and 
face-to-face modes, a hybrid course can also provide a layered approach where the same material 
is delivered in different modes—both online and face-to-face [2]. 

One of the reasons for moving course content online is the potential for making use of a learn¬ 
ing management systems (LMS). These software-based systems can not only display web-based 
content, but serve to manage and automate many of the administrative functions of a course [9]. 
This automation of administrative functions lowers the barriers to using ongoing formative and sum- 
mative assessments where the grading is partially or completely automated by the system [10,11]. 
As such, textbook or lecture content that was previously only assessed at a mid-term or final exam 
can now be reinforced though ongoing formative assessments in the form of weekly quizzes. For 
many of the reasons listed above, both students and teachers have indicated the perceived value 
of the learning tools made available in online courses [12]. 

In summary, newer web-based LMS systems have provided flexible options for componentized 
delivery of engineering course resources in the media and format that best suits learning outcomes 
and student acceptance, especially the delivery of material that was historically provided in a lec¬ 
ture setting. A hybrid course can, ideally, offer the best of face-to-face and online by selecting the 
delivery method best tuned for the learning goals and preferences of the students [13, 14]. A re¬ 
cent, large scale meta-analysis of educational research studies of online instruction [15] noted that, 
overall, instruction combining online and face-to-face elements had a larger advantage relative to 
purely face-to-face instruction than did purely online instruction. It also noted that this difference 
may not be necessarily rooted in the media used per se, but instead may reflect differences in a 
combination of content, pedagogy and learning time. That is, what is important is not only what 
is provided to students and how it was structured in the learning environment, but also how much 
time the students spent with it. 
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This finding points up the fact that online learning can be enhanced by giving learners control of 
their interactions with media, prompting learner reflection. Studies [15] indicate that manipulations 
that trigger more learner engagement through learner reflection and self-monitoring of understand¬ 
ing are effective when students pursue online learning as individuals. However, the meta-analysis 
by Means and colleagues [15] goes on say that elements such as video or online quizzes are not 
necessarily utilized as much as one would hope. The practice of providing online quizzes, in and of 
itself, does not seem to be more effective than other tactics such as assigning homework. 

Findings by Means and others raises interesting questions concerning how best to determine 
what resources to provide to students in a hybrid course and how to structure them so they will 
be utilized effectively in learning. Clearly, for instructional materials to support learning, they have 
to be used by students. It is not surprising that DeNeui and Dodge [16] found a significant positive 
partial correlation between overall LMS usage and student exam scores. A study by Hensley [10], in 
a comparison of purely online, hybrid, and face-to-face versions of the same pair of courses, showed 
significantly higher final grades for a purely online course over both the face-to-face and hybrid. 
However, the differences may have been the result of the removal of some of the motivating fac¬ 
tors to make use of the online resources in the hybrid format compared to when all of the material 
could only be accessed online. Previous work by the authors [17] has shown that, when used, online 
materials such as weekly assessments provided motivation to study the textbook material. 

The Graphic Communications faculty at North Carolina State University began offering a blended 
or hybrid version of their introductory engineering graphics course in the Fall 2007 semester [18, 
19]. The move was made in response to both budgetary pressures and a desire to provide a richer, 
more consistent presentation of what had traditionally been the lecture component of the course. 
The move to a hybrid format also provided the opportunity to provide a richer set of formative as¬ 
sessment opportunities and a centralized repository for online tutorials that were used in the lab 
component of the course. This move meant that one, two-hour meeting per week (primarily devoted 
to lecture on the textbook material) “went online.” The online portion of the course consisted of 
units (Figure 1) where students could watch streaming media of textbook lectures (Figure 2), solid 
modeling demonstrations (Figure 3), and sketching demonstrations (Figure 4). The online units 
also included weekly quizzes on the textbook material (Figure 5). This hybrid format still included 
a two-hour face-to-face meeting each week where faculty introduce the main concepts for the unit, 
answer questions about solid modeling and sketching activities, and check some homework. 

In order to better understand the impact of this marked change in course structure, a design- 
based study [20, 21] was instituted as part of an ongoing continuous improvement of instruction 
[22]. The goal was to carefully and systematically study student learning outcomes and attempt 
to understand how changes in course structure (and resulting instructor and student behaviors) 
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Unit 2-3D Form & Shape Description 

gL LESSON 4 - PROJECTION THEORY 

[Hi L4 TX Video - Material from Chapter 5 - 19:47 
[H[ L4 TX Video - Material from Chapter 7-6:16 
p[ Lesson 4 Assessment 

|f?[ Lesson 4 SolidWorks Modeling Activities 

Lesson 4 Modeling Assignment 

Hi L4 SW Video - Modeling the DRYER CLIP (Figure 5.145, page 289) - 12:56 
[Hi Lesson 4 Sketching Activities 
& Lesson 4 Sketching Assignment 
[Si L4 SK Video - ORT 010 - 5:53 
mi L4 SK Video - ORT 040 - 13:27 
[El L4 SK Video - ISO 020 - 12:31 

Figure 1: Example of an Online Unit for the Course. 


• Given these two 
views, can you 
form a mental 
image of the 3D 
assembly? 

• If you were given 
all of the parts, 
could you put them 
together using this 
drawing? 






Figure 2: Examples of Online Lecture Media. 


were linked to these outcomes. As is the nature of design-based research, this process is iterative 
with the analytic outcomes from each semester feeding into modifications of the course for the 
upcoming semester. 

Self-report data collected in the Fall 2007 and Fall 2008 semesters indicated general satisfaction 
with new course design. In addition, the collected data showed a correlation between performance 
on these weekly assessments and the final course grade as well as providing motivation to study 
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Figure 3: Example of Online Solid Modeling Demonstration. 


the textbook material [17]. One interesting finding from this initial research was self-report data on 
how the online resources were being utilized by the students. Streaming media presentations of the 
textbook material, solid modeling demonstrations, and sketching demonstrations were organized 
on course web pages. Students could navigate through the pages in any order. Each week students 
also were asked to complete an online assessment or quiz in the LMS. Post-course surveys were used 
to get feedback from students about how they used the online materials. In the Fall 2007 classes, 
students reported 19 different resource utilization sequence strategies for completing the material 
related to the textbook. 

For the spring 2009 semester, all online materials were moved to the Moodle LMS. By placing 
materials within Moodle, the authors could better track how students were navigating through the 
course. While the self-report data from students on how they used the online materials collected in 
our previous studies was insightful, it still suffered from students needing to accurately recall what 
instructional resources they used and in what order. While students may have been able to recall 
how they were making use of the instructional resources at the end of the semester, there were 
questions as to whether they could accurately recall what their pattern of use at the beginning of 
the semester. In addition, the survey did not attempt to gather information on whether this pattern 
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Which type of projection is indicated by the following symbol? 

9 

Marks: 

1 



Choose O a. Convergent projection 
O b. First-angle projection 

answer. 

O c. Perspective projection 
O d. Third-angle projection 


Figure 5: Example of an Online Assessment Item in the Course. 
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of use changed over the course of the semester. The online logging capabilities of Moodle allows 
the accurate tracking of the online resources students accessed and how these patterns may have 
changed over the course of the semester [23]. Online usage logs have been used in other studies 
[23-25] to better understand how online resources were being used and how these usage patterns 
might relate to learning outcomes. Of continued interest was whether the resources provided to the 
students for reviewing the textbook content—formatively assessing their knowledge, and therefore 
reflect on their understanding—was being used and what the relationship was between their use 
and exam scores (the summative assessment for the course). This type of analysis would provide 
greater insight as to the role of online resources in supporting student learning over of the course 
of the semester. 

In the design of the hybrid course, suppositions were made both as to what online resources 
would be of use to students in learning the course content and how these materials would be used. 
These suppositions were arrived at both from the prior research noted above and from our ongoing 
design-based study with this course. We were interested in testing these suppositions by looking at 
the use of the video-based PowerPoints™ (called Video in our analysis) and online quizzes (called 
Quiz) and whether there was a relationship between this usage and their final exam (called Exam). 
These suppositions were: 

1. Students should be reading the assigned textbook materials, reviewing the materials with the 
Video presentations and then testing their knowledge with the Quizzes. 

2. The Quizzes serve as a formative assessment of how well they have retained key concepts 
and terms. Since they can take the Quiz twice, they can also use the Quizzes to help cement 
material and to point to gaps that they can then review. 

3. Since the Quizzes count for 10% of their grade, there is an incentive to review materials (text¬ 
book and/or Video) prior to taking the quiz. 

4. The Video is arguably a better way to enhance learning of textbook material than the quizzes, 
and watching the Video then taking the Quiz is best. 

5. Ongoing use of Video and Quizzes will help prepare students for the summative assessment 
(final exam). 

6. With a total of nine possible units with Video and Quiz combinations, examining material from 
Units 1, 5 and 9 would give us a good look at student instructional resource usage behaviors 
at the beginning, middle, and end of the semester. 

7. Students use of resources would change over time, with the highest level of conformance to 
the ideal (using all available resources) being at the beginning of the semester and then tailing 
off as the semester progressed. 

Using these suppositions, the following research questions and hypotheses were explored: 


WINTER 2011 


7 






ADVANCES IN ENGINEERING EDUCATION 

Online Resource Utilization in a Hybrid Course in 

Engineering Graphics 


Q1: What is the relationship between the number of quiz attempts and final exam score? 

HI: Number of units in which there was at least one quiz attempt will correlate with the final 
exam score. 

Q2: What is the relationship between the accessed and used resources (video, quizzes) and final 
exam score? 

H2a: Use of the Video in Units 1, 5, 9 would predict Exam score 

H2b: Use of the Video and then taking the Quiz in Units 1, 5, 9 would predict Exam score 

Q3: How does access and use of resources change over time? 

H3a: Use of the Video would change between Units 1, 5, 9 

H3b: Use of the Video and then taking the Quiz would change between Units 1, 5, 9 


METHODOLOGY 

During the fall 2009 semester, three blended learning sections of GC120 were studied. All online 
materials were accessed only through Moodle. An additional difference in these sections from previ¬ 
ous semesters was the enrollment for each section was set at a maximum of 60 students instead of 
24 in previous semesters. Tables 1-3 display the demographic data of the students in the blended 
sections of the course. 

As in previous semesters, students were required to view and complete online materials on a 
weekly basis. Materials were organized into 12 weekly online units. Each unit consisted of streaming 
media presentations of the textbook material, streaming media SolidWorks™ demonstrations, and 
streaming media sketching demonstrations. Students also had to complete a 10-20 question Quiz 
in Units 1 through 5 and 8 through 11 as a check of their textbook knowledge (Units 6, 7 and 12 did 
not have assigned textbook material). They were given two attempts at each assessment, if needed. 


Section 

Freauencv 

Percent 

003 

55 

34% 

004 

56 

34% 

005 

53 

32% 

TOTAL 

164 

100% 


Table 1: Enrollment Per Blended Section of GC120. 
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Year 

Frequency 

Percent 

Freshmen 

4 

2% 

Sophomore 

115 

70% 

Junior 

29 

18% 

Senior 

16 

10% 

TOTAL 

164 

100% 


Table 2: Academic Year. 


Major 

Freauencv 

Percent 

Aerospace Engineering 

17 

10% 

Civil Engineering / Construction 

45 

27% 

Management 



Mechanical Engineering 

44 

27% 

Other Engineering Majors 

29 

18% 

Education 

7 

4% 

First Year College 

8 

5% 

Other Majors 

14 

9% 

TOTAL 

164 

100% 


Table 3: Academic Major. 


For each assessment, there was paired a streaming video of a voiced-over PowerPoint™ presentation 
of the key concepts of the required textbook readings for the week (the Video). 

Moodle logging tools were used to record when each of these resources were accessed for each 
student. Data was extracted from the logs for each student enrolled in the course. A purposeful 
sample of Moodle units was examined, with data for Units 1, 5 and 9 used for this study. Students 
who did not take the final exam in the course were excluded from the sample. 


RESULTS 

Tables 4-6 display the order in which students completed the Videos and the Quizzes for these 
three units. Because students could take the Quizzes twice, access the Video as many times as they 
wanted, and use both these resources in any order they wanted, there were a number of possible 
permutations of how students accessed these materials. 

During Unit 1 the most popular strategy for students was viewing the streaming media video and 
then taking the online assessment twice (38%). As stated earlier, students were allowed a maximum of 
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Order of Activities 

Frequencv 

Percent 

VQQ 

63 

38% 

VQ 

52 

32% 

V 

16 

10% 

QQ 

11 

7% 

Q 

7 

4% 

VQVQ 

7 

4% 

Did not view media or complete assessment 

5 

3% 

QQV 

2 

1% 

VOV 

1 

1% 

TOTAL 

164 

100% 


Q: Attempt at online assessment/quiz. 

V: Viewed streaming media video of textbook material. 


Table 4: Order Students Completed Moodle Activities in Unit 1. 


Order of Activities 

Frequency 

Percent 

Q 

45 

28% 

VQQ 

42 

26% 

QQ 

35 

21% 

VQVQ 

13 

8% 

Did not view media or complete assessment 

11 

7% 

VQ 

10 

6% 

V 

4 

2% 

OVO 

4 

2% 

TOTAL 

164 

100% 


Table 5: Order Students Completed Moodle Activities in Unit 5. 


two attempts at each assessment. The second most popular strategy in Unit 1 was viewing the stream¬ 
ing media video and then taking the online assessment once (32%), followed by only viewing the video 
(10%). In Unit 5 the most common strategies for students were taking the online quiz without viewing 
the streaming media video (28%), viewing the streaming media video and then taking the online as¬ 
sessment twice (26%), and taking the assessment twice without viewing the streaming media video 
(21%). The top three strategies used in Unit 9 were viewing the streaming media video and then taking 
the online assessment twice (27%), taking the assessment twice without viewing the streaming media 
video (23%), and taking the assessment once without viewing the streaming media video (20%). Tables 
4-6 revealed a total of ten different possible sequences: 
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Order of Activities 

Frequencv 

Percent 

VQQ 

44 

27% 

QQ 

38 

23% 

Q 

32 

20% 

Did not view media or complete assessment 

25 

15% 

VQ 

12 

7% 

V 

5 

3% 

VQVQ 

4 

2% 

QQV 

3 

2% 

ovo 

1 

1% 

TOTAL 

164 

100% 


Table 6: Order Students Completed Moodle Activities in Unit 9. 


Watched Video (Vyes)? 

Sequences 

No (0) 

Q, QQ, Nothing 

Yes (1) 

V, VQ, VQQ, VQV, VQVQ, QQV, QVQ 


Table 7: Sequence Grouping based on Watching the Video. 


Watched Video and took Quiz? (VQyes) 

Sequences 

No (0) 

Q, QQ, v, QQV, Nothing 

Yes (1) 

VQ, VQQ, VQVQ, VQV, QVQ 


Table 8: Sequence Grouping based on Watching the Video and Taking the Quiz. 


Mean 

81.76 

Std. Deviation 

8.76 

Minimum 

54 

Maximum 

98 


Table 9: Exam Score Descriptive Statistics. 
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Total Quiz 

Exam 




Attempts 

Score 

Spearman's rho 

Total 

Correlation Coefficient 

1.000 

.181* 


Quiz 

Sig. (1-tailed) 


.012 


Attempts 

N 

164 

156 


Exam 

Correlation Coefficient 

.181* 

1.000 


Score 

Sig. (1-tailed) 

.012 




N 

156 

156 


*. Correlation is significant at the 0.05 level (1-tailed). 


Table 10: Correlations - Total Quiz Attempts x Exam Score. 


Change Statistics 


Model 

R 

R 

Square 

Adjusted R 
Square 

Std. Error of 

the Estimate 

R Square 
Change 

F 

Change 

dfl 

df2 

Sig. F 
Change 

Durbin- 

Watson 

1 

.213 

.045 

.039 

8.603 

.045 

7.319 

1 

154 

.008 


2 

.223 

.050 

.037 

8.610 

.005 

.729 

1 

153 

.395 


3 

.237 

.056 

.037 

8.610 

.006 

.997 

1 

152 

.320 

1.950 


Model 1 - Predictors: (Constant), Vyes5 
Model 2 - Predictors: (Constant), Vyes5, Vyes9 
Model 3 - Predictors: (Constant), Vyes5, Vyes9, Vyesl 
Dependent variable for all models: Exam 

Table 77; Vyes X Exam Regression Mode! Summary. 


1. V 

2. VQ 

3. VQQ 

4. VQV 

5. VQVQ 

6. QQV 

7. QVQ 

8. Q 

9. QQ 

10. Nothing 
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Model 


Sum of Squares 

df 

Mean Square 

F 

Sig. 

1 

Regression 

541.605 

1 

541.605 

7.319 

.008 


Residual 

11396.620 

154 

74.004 




Total 

11938.224 

155 




2 

Regression 

595.623 

2 

297.811 

4.017 

.020 


Residual 

11342.602 

153 

74.135 




Total 

11938.224 

155 




3 

Regression 

669.570 

3 

223.190 

3.011 

.032 


Residual 

11268.654 

152 

74.136 




Total 

11938.224 

155 





Model 1 - Predictors: (Constant), Vyes5 
Model 2 - Predictors: (Constant), Vyes5, Vyes9 
Model 3 - Predictors: (Constant), Vyes5, Vyes9, Vyesl 
Dependent variable for all models: Exam 

Table 12: Vyes X Exam ANOVA d . 


Change Statistics 


Model 

R 

R 

Square 

Adjusted 

R Square 

Std. Error of 

the Estimate 

R Square 
Change 

F 

Change 

dfl 

df2 

Sig. F 
Change 

Durbin- 

Watson 

1 

.204 

.042 

.035 

8.619 

.042 

6.699 

1 

154 

.011 


2 

.226 

.051 

.039 

8.605 

.009 

1.513 

1 

153 

.221 


3 

.228 

.052 

.033 

8.630 

.001 

.123 

1 

152 

.726 

1.949 


Model 1 - Predictors: (Constant), Vyes5 
Model 2 - Predictors: (Constant), Vyes5, Vyes9 
Model 3 - Predictors: (Constant), Vyes5, Vyes9, Vyesl 
Dependent variable for all models: Exam 

Table 13: VQyes X Exam Regression Model Summary d . 


These sequences can be organized into two groups based on whether the sequence included 
watching the Video (Vyes) (Table 7). 

A second way is to group those sequences that included both watching the Video (at least once) 
before taking the Quiz (at least once)—called VQyes (Table 8): 

This method of organizing the possible sequences of instructional resource utilization provides 
a means for answering the previously posed research questions and paired hypotheses. Answering 


WINTER 2011 


13 















ADVANCES IN ENGINEERING EDUCATION 

Online Resource Utilization in a Hybrid Course in 

Engineering Graphics 


Model 


Sum of Squares 

df 

Mean Square 

F 

Sig. 

1 

Regression 

497.640 

1 

497.640 

6.699 

.011 


Residual 

11440.585 

154 

74.290 




Total 

11938.224 

155 




2 

Regression 

609.665 

2 

304.833 

4.117 

.018 


Residual 

11328.559 

153 

74.043 




Total 

11938.224 

155 




3 

Regression 

618.835 

3 

206.278 

2.770 

.044 


Residual 

11319.389 

152 

74.470 




Total 

11938.224 

155 





Model 1 - Predictors: (Constant), Vyes5 
Model 2 - Predictors: (Constant), Vyes5, Vyes9 
Model 3 - Predictors: (Constant), Vyes5, Vyes9, Vyesl 
Dependent variable for all models: Exam 

Table 14: VQyes X Exam ANOVA d . 



Value 

No (0) 

Yes (1) 

Vyesl 

23 

141 

Vyes5 

91 

73 

Vyes9 

95 

69 


Table 15: Frequency Counts for Vyes. 


Value 


No (0) 

Yes (1) 

VQyes 1 

41 

123 

VQyes5 

95 

69 

VQyes9 

103 

61 


Table 16: Frequency Counts for VQyes. 
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these research questions, then, provides a means of comparing our suppositions about optimal usage 
of the resources against actual usage. The research hypotheses are used to organize the quantitative 
results, with these results relative to the research questions synthesized in the Discussion. Students’ 
final exam score (Exam) is used as part of analyses for all of the research hypotheses. The Exam 
variable descriptive statistics are provided in Table 9 

HI: Number of units in which there was at least one Quiz attempt will correlate with the final 

Exam score. 

A Kolmogorov-Smirnov test of normality showed that the distribution of number of quiz at¬ 
tempts was non-normal (p < .001), therefore a Spearman correlation was used (Table 10). There 
was a significant correlation ( rho =.181, p < .05) between the number of quiz attempts and final 
exam score. 

H2a: Use of the Video (Vyes) in Units 1, 5, 9 would predict Exam score 

H2b: Use of the Video and then taking the Quiz (VQyes) in Units 1, 5, 9 would predict Exam 

score 

Initial analysis of Vyes indicated that Video use in Unit 5 had the highest correlation with Exam, 
with Unit 9 next and then Unit 1. Therefore a forward step-wise regression was modeled, adding 
Units 5, 9 , and 1 in that order (Tables 11 and 12). The model shows that Vyes5 was the best predictor 
of Exam score and that Vyes9 and Vyesl did not add anything to the prediction. Results of Durbin- 
Watson test, tolerance and VIF indicate that assumptions of residual independence and collinearity 
were not violated. 

A parallel examination of Video plus Quiz (VQyes) revealed very similar results, with VQyes5 
being the best predictor and VQyes9 and VQyesI adding very little to the model (Tables 13 
and 14). 

H3a: Use of the Video (Vyes) would change between Units 1, 5, 9 

H3b: Use of the Video and then taking the Quiz (VQyes) would change between Units 1, 5, 9 

Since Vyes and VQyes can be considered repeated measures and that these measures can also 
be considered bivariate categorical data, a Cochran’s Q test was conducted to test the null hy¬ 
pothesis that there was no change in the categorical variable (Vyes or VQyes) over time (Units 1, 5, 
and 9). Separate tests were run for Vyes (Table 15) and VQyes (Table 16). Both tests for both Vyes 
(Q = 101.28; p < .001) and VQyes (Q = 74.17; p < .001) were significant. Examination of the counts 
showed that a majority of students conformed to the ideals (view the Video then take the Quiz) at 
the beginning of the semester (Unit 1) but that this pattern reversed by Unit 5, dropping off slightly 
more by Unit 9. 
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DISCUSSION 

This study was part of an ongoing design-based study examining how the move to a hybrid 
course format impacted engineering graphics students’ learning. Confirming the self-report find¬ 
ings in previous studies of our online course, students used a diversity of approaches to making 
use of the online resources. While a number of students followed the explicitly recommended 
order of material use (i.e., view the streaming video before attempting the quiz), many students 
took alternate approaches to usage and usage order. Research Question 1 was focused on whether 
use of the formative assessment resource, the online quizzes, was positively correlated with per¬ 
formance on the final exam (the summative assessment of the same course content). The results 
seem to indicate that the instructors had done a good job aligning the formative and summative 
components and that usage of this formative assessment resource was associated with higher 
scores on the final exam. 

With the weekly online quiz assessments only worth a total of 10% of students’ final grade and 
the midterm and final worth collectively 40% of the grade, it was our assumption that students 
primarily saw the value in the weekly assessments as a reflection tool preparing them for the larger 
summative assessments of the midterm and final. The data collected seem to support the conclusion 
that those students who attempted more weekly assessments (and/or made use of the streaming 
videos) did better on the final exam. 

Research Question 2 explored the supposition that use of the video-based PowerPoint™ presenta¬ 
tions, in combination with the online quizzes, would predict the final exam score. Models exploring 
the Video only and the Video in conjunction with the Quiz showed themselves significant but rather 
modest predictors of final exam scores. Interestingly, it was the Quiz taken just before the midterm 
(Unit 5) that had the highest predictive power. Perhaps that during the first week of class (Unit 1), 
students all conformed more or less to expectations articulated by the instructor. By the end of 
the course (Unit 9), there were usual pressures related to the end of the semester. These different 
external pressures at the beginning and the end of the semester may have altered the relationship 
between resource usage and exam score. 

Findings related to Research Question 3 also support the role of external pressures that seem to 
weigh on students at different points in the semester. Students did conform most closely to the ideal 
as articulated by the instructors at the beginning of the semester, but fell off significantly in their 
use of these resources by the end of the semester. Interestingly, this fall-off was already evident by 
mid-semester. While students may have determined themselves that the Video and Quizzes were 
not of enough value to spend time on them, the results of the correlational and regression analy¬ 
ses seems to indicate that they are of value in improving performance on the final exam. It may be 
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instead, that building pressures on a student’s time more than a conscious decision to forgo these 
resources led the students to stop using them. 

The findings from this study have a number of implications for other instructors implementing 
purely online or hybrid engineering courses. The flexibility provided by LMSs means that students 
can be given great latitude as to when they access materials and how they use them as part of 
their overall learning strategy for the course. Our studies show that when provided flexible access, 
students will use the resources (or not) in a wide variety of sequences to serve what they perceive 
is their learning needs. Our current study also demonstrates that how a student accesses these 
resources changes over the course of the semester, probably due both to their ongoing experience 
with the course and the changing time pressures that normally build over the course of an academic 
semester. We clearly saw a drop-off in the use of resources over the course of the semester. It may 
be that if the weekly quizzes were worth more of the final grade, they would have been taken more 
seriously. However, the higher percentage weighting on the quizzes would have made them less of 
a formative, self-study tool for the students and more of a high-stakes mini-summative assessment. 
If an instructor has a strong preference as to the sequence in which resources are accessed, they 
may want to investigate the tools available in their particular LMS to control the order in which ma¬ 
terials are accessed. Similarly, “time window” tools can be used to control when items are available. 
For the course in this study, weekly quizzes were only available for a few days during the week of 
that particular lesson, preventing students from cramming for multiple quizzes closer to exam time. 
All of this needs to balanced against the general student preference for flexibility in making use of 
instructional resources. 


CONCLUSION 

Probably the most important finding of this study is that the logging tools provided in Moodle 
provides a powerful way for instructors to gather and analyze data on how students make use of 
the resources provided online. Future work will include exploration with new ways these tools can 
be deployed to support course development, teaching and learning. The researchers found it much 
easier to chart student trends and performance in a more accurate way than in past incarnations 
of this course. This provides instructors with better information to use in the redesign of course 
materials for the future with the desire to aid the students more in learning the material. Since the 
production of the multimedia learning resources is a labor-intensive practice, this formative data 
for the instructor provides valuable evidence as to whether such material is being used by students 
and whether it provides real educational value. Another possible avenue for research would be to 
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see if demographic variables, such as year in school or gender, predict different patterns of usage. 
Just as usage patterns of course resources seem to vary over time, they may also vary between dif¬ 
ferent student sub-populations. In the short-term, resource (especially quiz) utilization monitoring 
can help alert instructors to students whose use of the course materials may have changed markedly 
during the course of the semester. 

One of the limitations of this study was the granularity of the log data. The researchers were able 
to identify if a resource, like a video, was accessed but not how long it was viewed, or if segments 
were replayed. Future research work will look at developing methods for more fine-grained analysis 
of log data. This work would include more data points over the semester to better understand trends, 
analysis of first versus second tries on quizzes, and the use of SCORM-compliant learning resources 
[26] in conjunction with Moodle that allow richer data collection usage (e.g., how long did they view 
a video and how many times did they stop and start it). In addition, this work will also move into 
analyzing resources related to the laboratory portion of the course. This study also had the general 
limitation as to how both formative and summative assessments were conducted—that is, primarily 
through multiple choice questions. While the lab portion of the course created CAD files of student 
work that provided insight into learning the applied portion of the course, we are continuing to 
look for assessment tools to use in conjunction with the textbook content that provide richer data. 
Again, log data-in this case, how students might interact with queries and probes provided by the 
LMS—may help provide a mechanism for this. 

In conclusion, this study demonstrated that LMSs like Moodle both provide expanded flexibility 
in the types of learning resources provided to students and the tools provided to instructors for 
monitoring their use. An important finding of this study was that use of these resources changed 
over the course of the semester and that monitoring these patterns of change may help alert in¬ 
structors to emerging learning patterns, which may impact students’ mastery of course content. 
Further research in this area that might enhance the power of this monitoring would be to focus 
both on increasing the granularity of the log data collected on resource utilization and expanding 
the types of assessments used to assess student learning. 
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